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Abstract 

An area of increasing interest within AI and Robotics is the integration of techniques from 
both fields to the problem of controlling autonomous systems. Space-based systems, such as 
NASA’s EVA Retriever, provide complex, realistic domains for this integration research. Space is a 
dynamic environment, where information is imperfect, and unexpected events are commonplace. 

As such, space-based robots need low level control for collision detection and avoidance, short- 
term load management, fine-grained motion, and other physical tasks. In addition, higher level 
control is required to focus strategic decision making as missions are assigned and carried out. 
Throughout the system, reasoning and control must be responsive to ongoing change taking 
place in the environment. 

This paper reports on current MITRE research aimed at bridging the gap between high level 
AI planning techniques and task-level robot programming for telerobotic systems. Our approach 
is based on incorporating situated reasoning into AI and Robotics systems in order to coordinate 
a robot’s activity within its environment. Thus, the focus of this research is on controlling 
a robot embedded in an environment, as opposed to the generation and execution of lengthy 
robot plans. We present an integrated system under development in a “component maintenance” 
domain geared towards repair and replacement of Orbital Replacement Units (ORUs) designed for 
use aboard NASA’s Space Station Freedom. The domain consists of a component-cell containing 
ORU components and a robot (manipulator and vision system) replacing worn and/or failed 
components based on the collection of components available at a given time. High level control 
reasons in “component space” in order to maximize the number operational component-cells 
over time, while the task-level controls sensors and effectors, detects collisions, and carries out 
pick and place tasks in “physical space.” Situated reasoning is used throughout the system to 
cope with, for example, non-deterministic component failures, the uncertain effects of task-level 
actions, and the actions of external agents operating in the domain. 

*This work is funded by MITRE- Sponsored Research Projects 97060 and 97140. The authors wish to thank 
MITRE for its ongoing support of this work. Email address: sanborn@ai.mitre.org. 
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1 Introduction 


There is a resurgence of interest within the 
AI and Robotics communities in integrated ef- 
forts leading to the development of robust, au- 
tonomous systems for use in dynamic, uncertain, 
and unpredictable domains. NASA in particu- 
lar has several efforts underway, including its 
Systems Autonomy Technology Program (SATP), 
a ten year program to establish NASA as a 
world leader in intelligent autonomous systems 
research and development, the EVA Retriever, 
and the Mars Rover project. These programs 
are aimed at addressing two issues in space ex- 
ploration: 

1. For manned missions, human EVA is danger- 
ous, expensive, and time-consuming. 

2. For unmanned missions, signal delay times 
require autonomous control throughout 
non-trivial time intervals. 

One of the major hurdles in building autonomous 
systems is the integration of off-line deliberative 
reasoning (e.g., task planning, route planning, 
and resource allocation, etc.) with real-time sit- 
uated control (e.g., collision and obstacle avoid- 
ance, path expansion, load management, cali- 
bration from landmarks, etc.). The former type 
of reasoning is goal- directed , and has led to the 
development of several constraint-posting plan- 
ners (e.g., [Ste81], [Wil84], [Cha85]) generating 
plans to satisfy multiple goals. The latter type 
of reasoning is event-driven , in that responses to 
existing, perceived, or projected situations are 
required in order to maintain overall system in- 
tegrity. AI research has begun to address situ- 
ated reasoning, as well its integration with off- 
line plan generation (see, for example, [Kae86], 
[GLS87], [AC87], [Dea87], and [SH88]). 


This paper presents the initial results of a com- 
bined MITRE research effort integrating AI plan- 
ning and situated reasoning techniques with 
task-level robotics and perception for space- 
based autonomous systems. The long-term goal 
of this research is to integrate off-line planning, 
situated reasoning, and sensor /actuator subsys- 
tems across various levels of abstraction in order 
to provide both the reactive behavior necessary 
for survival in realistic environments, and the in- 
trospective reasoning required to carry out delib- 
erate tasks and achieve desired goals. The work 
presented in this report lays the groundwork for 
this long-term goal by providing an integrated 
situated reasoning and task-level control archi- 
tecture, as well as a system operating in a re- 
alistic application domain. Examples from this 
domain are used throughout the paper to illus- 
trate the approach. 

2 A Component Repair and 
Maintenance Domain 

One of the many application areas for space- 
based autonomous systems is routine extra- 
vehicular maintenance. By “routine mainte- 
nance,” we refer to a general class of situations 
in which components of a system are scheduled 
for maintenance (as determined by expected life- 
time) and are also tended to when they fail un- 
expectedly. In such an application, a robot must 
allocate available resources (spare parts, or mod- 
ular components such as ORUs) in order to max- 
imize the overall operating status of a collection 
of components. 

The “routine repair and replace” domain shown 
in Figure 1 captures this idea. It consists of a 
robotic manipulator, vision system, a component 
workcell, and a collection of components. The 
workcell is an N x M array of compartments, 
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Workcell 



(a) Physical Space (b) Component Space 

Figure 1: A Simple Component Repair and Maintenance Domain 


each of which may be “filled” by components 
of various types. In this way, the workcell may 
be thought of as a modular “breadboard” into 
which components are inserted to become oper- 
ational. Each workcell compartment is labelled 
according to the types of components that may 
fill it. A component is said to be acceptable to a 
compartment whenever it may be used to fill that 
compartment. Finally, an expected lifetime (the 
mean time to failure , MTTF) is associated with 
each type of component in order to model both 
routine and unexpected component failures. 

In the current system, the workcell is mod- 
elled by a “bin” on a tabletop, with different 
shaped objects (cylinders, rectilinear and trian- 
gular blocks, etc.) representing various compo- 
nent types (see Figure 1(a)). States of the do- 
main are subject to constant flux at the hands 
of external (human) agents, whose unanticipated 
actions may include adding, removing, moving, 
and breaking components. In addition, the fact 
that components may fail unexpectedly at any 


time also requires attention to the ongoing sit- 
uation. The choice of this repair and replace 
domain was influenced by the following consid- 
erations: 

1. The architecture should be realistically scal- 
able to handle any of a variety of repair and 
replace tasks to be performed in environ- 
ments characterized by dynamics and uncer- 
tainty (such as Space Station ORU replace- 
ment) . 

2. The scenario requires the integration of 
physical control (robotics) with high-level 
reasoning (Al). 

3. The hardware required for developing the 
testbed scenario was readily available. 

2.1 Physical vs. Component Space 

The reasoning required for successful operation 
in this domain falls into two classes: reason- 
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ing in physical space , and reasoning in compo- 
nent space. Physical space reasoning includes the 
planning, executing, and monitoring of collision- 
free paths for the manipulator and moved ob- 
jects, detecting obstacles, noticing objects when 
they are moved, and generally dealing with phys- 
ical aspects of the domain. The physical space 
reasoner used to control the manipulator and vi- 
sion system shown in Figure 1(a) is known as 
the Task-level Robot Programming System (TL- 
RPS). Component space reasoning includes allo- 
cating available components among empty com- 
partments, prioritizing replacement and repair 
tasks according to various heuristics, as well 
as reacting to unexpected component failures, 
moves, additions, and deletions. The Component 
Space Reasoner (CSR) provides this functionality 
for the component space corresponding roughly 
to Figure 1(b). 

2.2 Interface Language 

This section presents the communication speci- 
fication between TLRPS and CSR. The interface 
has been designed to distinguish between physi- 
cal and component space aspects of the domain. 


2.2.1 C SR— ►TLRP S 

Put_in object compartmenti : Move object from 
its present (table) location into the (empty) 
compartmenti. 

Put_at object x y : Place object on the table- 
top at TLRPS coordinates ( x y); object is as- 
sumed to be either on the table or within 
the workcell. 

Put_down object x y : Put down (held) object on 
the tabletop at TLRPS coordinates (zy). 


2.2.2 TLRPS— ►CSR 

Begin_update : Initiates an update of object 
and location information from TLRPS. Fol- 
lowed by one or more instance of: 

Delete object : object has been removed 

from the domain. 

Move object x y 9 : object now centered at 
(x y) rotated by 6 degrees. 

Add object x y 9 : object has appeared in the 
domain, centered at (x y) rotated by 0. 

End_update : Signals the end of the update. 

Failed-grasp object : TLRPS could not grasp 
object. 

Collision held-object objectJn.path : ob - 

jectJn.path prevents moving held^object. 

Unreachable job ject object : object cannot be 
reached in its current location. 

Unreachable .location x y : (x y) cannot be 
reached. 

3 Technical Approach 

This section describes the operation of TLRPS, 
CSR, and their integration in the repair and re- 
place domain. Since this domain is non-static, 
each system must cope with discrepencies be- 
tween anticipated and actual states of the do- 
main. For example, components may move from 
expected locations and may fail (or be broken) 
before their MTTF has elapsed. Since exter- 
nal agents may change the environment, nei- 
ther system can make accurate long-term projec- 
tions regarding future states. Rather, the system 
must optimize local behavior based on existing 
and projected states given the overall component 
maintenance goals. The top-level system archi- 
tecture is shown in Figure 2. 
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Figure 2: Top-level System Architecture 


3.1 Physical-Space Reasoning 

TLRPS resides on a Silicon Graphics IRIS 
4D/70GT workstation running the IGRIP 3D 
robotics modeling and simulation system from 
Deneb Robotics. The vision system consists of 
software provided by NASA running on an IBM 
PC-AT with added frame grabber and image 
processing boards from Data Translation. The 
robot currently in use is a Microbot Alpha I. 

A single camera with a fixed viewpoint is used to 
capture the layout of the robot’s workspace. The 
vision software in the PC-AT classifies the objects 
in the workspace according to its training data 
set corresponding to the physical component ob- 
jects. A workspace description containing object 
types and their locations and orientations is then 
sent to the IRIS. Image processing software on 
the IRIS interprets the workspace description on 
each cycle and modifies the world model accord- 
ingly, generating CSR update messages, as well as 
updating the IGRIP simulation’s 3D graphic dis- 


play of the robot and the workspace. Users may 
interact directly with TLRPS by entering task- 
level commands via pop-up menus, or turn con- 
trol over to CSR. When a task-level command 
is received by TLRPS (whether from CSR or a 
human user), the command is simulated in 3D 
graphics and executed by the robot. The simu- 
lation runs one step ahead of the actual robot 
execution, checking for possible collisions and 
out-of-reach conditions. If such a condition is 
detected, TLRPS performs error recovery opera- 
tions, registering the simulation and the robot 
to a safe configuration. An exception-dependent 
error message is generated and sent to CSR when 
it is controlling TLRPS. 

3.2 Component-space Reasoning 

CSR is divided into two main modules: an agenda 
manager for prioritizing tasks and issuing com- 
mands to TLRPS, and a collection of objects cor- 
responding to the various physical objects in the 
domain at a given time. CSR’s top-level control 
loop is 

(0) get component types used 

(1) await TLRPS update 

(2) process TLRPS update 

(3) if there is an executable task 

(4) then issue it to TLRPS 

(5) goto (1) 

The initialization step (0) determines compo- 
nents type definitions and their associated pa- 
rameters. This is accomplished by consulting a 
file provided before startup. Once this informa- 
tion has been processed, CSR enters its main con- 
trol loop. The first stage of this loop (Step 1 
above) simply puts CSR into a wait state un- 
til an update from TLRPS is available. Recall 
that these updates consist of a collection of Add, 
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Delete, and Move messages. The second stage 
(Step 2) processes the update’s content. Dur- 
ing this stage, messages are sent to existing CSR 
objects mentioned in the update; the message 
sent depends on what the update to the object 
happens to be. During the processing of these 
messages, objects may request that the agenda 
manager generate new tasks to, or remove ex- 
isting tasks from, its collection of tasks. In the 
final stage (Steps 3 and 4), the agenda manager 
prioritizes its collection of tasks and, if any are 
executable, issues a command to TLRPS to carry 
out the most important executable task. Within 
CSR, a task is one of: 

routine_replace component compartmenti : 
replace worn (but operating) component 
with an existing new component acceptable 
to compartmenti . 

immediatejrepair component compartment t - : 
replace failed component . 

await_component compartmenti • fill 

compartmenti when an acceptable compo- 
nent is added. 

install.component component compartmenti : 
put component into compartmenti and start 
it operating. 

deinstall_component component 

compartmenti : stop operating component 
and remove it from compartmenti. 

move .component component z y : move compo- 
nent from its current location to (x y). 

3.2.1 Active Objects 

Most of the reasoning done by CSR is triggered 
during update processing. For each physical ob- 
ject in the domain, CSR generates a component 


object in its component space. The workcell, and 
each of its compartments, is also represented as 
an active CSR object. Updates from TLRPS relat- 
ing to physical objects are then translated into 
messages to CSR objects, which may take var- 
ious update-dependent actions. This approach 
associates reasoning with changing information 
at the object level, rather than on the more 
traditional rule-interpreter/dat abase approach. 
As such, reasoning and control are modified by 
changing objects’ responses to messages, rather 
than by the actions contained in rule conse- 
quents. 

To illustrate this approach to object modelling 
and situated reasoning, we follow the process- 
ing of CSR’s initial update. This update con- 
sists of a collection of Add messages from TL- 
RPS. The first such message processed corre- 
sponds to the workcell. On encountering this 
message, CSR generates a workcell object, an ob- 
ject for each of the workcell’s compartments, and 
initializes their parameters. Following this, a 
component object is created for each added com- 
ponent. As part of its Add message processing, 
the component checks to see whether or not it 
lies within some compartment. If not, its status 
is “new.” Otherwise, the compartment object is 
consulted to determine whether or not the com- 
ponent should be accepted. If so, the compart- 
ment “installs” the component, by setting the 
component’s status to “operating,” and sched- 
ules a routine ^replace task for sometime in the 
future, depending on the MTTF of the newly in- 
stalled component. If the compartment does not 
accept the component, it attempts to have the 
offending component removed by generating a 
move .component task. 

Once this initial update has been processed, 
CSR knows which compartments contain op- 
erating components, which need to be filled, 
as well as which components are available in 
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the domain. Empty compartments generate 
await.component tasks and compete for newly 
arriving components according to several crite- 
ria, including (1) how long the compartment 
has been empty, and (2) the number of differ- 
ent types of components the compartment ac- 
cepts. The next section discusses the use of these 
heuristics in prioritizing tasks. 

3,2,2 Tasks and the Agenda Manager 

In order to take action, component and compart- 
ment objects generate tasks which are added to 
CSR’s agenda : a simple partially ordered set of 
tasks. All tasks have the following properties, 
which are used in determining their relative im- 
portance and/or execution status: 

status : one of 

: pending : not yet ready for execution, 

: executable : ready for execution, 

: active : in progress, or 

:done : finished, does not indicate success 
or failure. 

priority : a measure of the task’s importance, 
one of 

: normal : a routine task, such as 

routinejreplace, 

:asap : a non-routine task, such as 

immediatejrepair, or 

:now : highest priority tasks, such as 

await.component. 

resource .measure : measure of difficulty in ob- 
taining resources for this task. 

timestamp : actual task instantiation time. 

supertask : associated parent task. 


subtasks : subtasks comprising an abstract 
task. 

Tasks fall into two classes: primitive tasks, 

corresponding directly to TLRPS commands, 
and abstract tasks, which have no analo- 
gous TLRPS command. Abstract tasks may 
have an associated test executed once per cy- 
cle whenever the task’s status is : pending. 
All tasks have an act , which is executed 
when an : executable task becomes : active. 
In the current system, install.component, 
deinstall.component, and move .component are 
primitive (corresponding to various types of 
put_in, put_at, and put_at commands, respec- 
tively); all other CSR tasks are abstract. 

In most traditional plan generation and execu- 
tion systems, a complete plan to achieve a goal 
is generated and then executed stepwise. An un- 
derlying assumption of this approach is that the 
world will behave as expected during plan exe- 
cution. If exceptions occur during execution, the 
usual recourse is to more planning. As an exam- 
ple, a “routine replace” operation on a compart- 
ment is normally composed of two steps: 

1. deinstall_component oldc compartmenti 

2. install.component newc compartmenti 

under the assumption that newc will be avail- 
able when step (2) is to be executed. How- 
ever, if newc is removed sometime during the 
execution of step (1), the resulting situation is 
the same as one in which compartmenti were 
simply waiting to be filled. Recall that in 
this case, the empty compartment generates an 
await .component task to find and then install a 
suitable component. 

In general, CSR uses component and compart- 
ment objects to assist in carrying out plans 
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whenever the domain is cooperative, but also en- 
sures that the appropriate behavior results when 
“assumptions” fail. Rather than planning steps 
for anticipated future states, CSR generates one 
step at a time, and uses feedback from the world 
to determine its next step. The definition for 
routine_replace looks like 

(deftask routine_replace 
(compartment 
component 
new_c omponent 
replace_time) 

:test (and (> (now) replace_time) 
new_c omponent ) 

: act (generate_task 

’deinstall^component 
component compartment) 

(reserve compartment 

new.c omponent) ) 

Notice that there is no mention of a task cor- 
responding to step (2) above in this defini- 
tion. If all goes well in the world, the compart- 
ment will generate an install.c omponent us- 
ing the new component, reserved for it by the 
routine_replace task, once the deinstall has 
been successfully carried out. However, if for 
some reason the new component is no longer 
available, the compartment simply generates an 
await .component, which searches for another 
suitable replacement. Since no assumption is 
made as to whether or not the new component 
will remain available, the appropriate response 
occurs in either case. This approach requires 
that CSR objects track their allocations (so that, 
for instance, the compartment can determine 
whether or not its reserved component is avail- 
able or not), but this is easily managed by in- 
forming objects of their allocations and taking 
appropriate action during update message pro- 
cessing. 


The task agenda is ordered in decreasing order 
of importance. This ordering is maintained by 
merging new tasks, and re-merging tasks when 
their parameters change, according to the follow- 
ing sequence of pairwise tests: 

1. status - executable > rpending 

2. priority - :now > :asap > : normal 

3. resources - prefer more constrained 

4. time - prefer older tasks 

5. arbitrary 

One case where re-merging is necessary is when 
an acceptable component becomes available to 
a pending await .component task. New com- 
ponents announce their availability by sending 
an advertise-available-resource message to 
the agenda manager, which in turn offers the new 
resource to pending tasks in decreasing order of 
importance. If a task allocates the available re- 
source, the resource is allocated to it, and it is re- 
merged into the agenda. When an available com- 
ponent is offered to an await .component task, 
it is allocated so long as it is acceptable to the 
task’s compartment. 

Another way an await .component task may find 
an acceptable component is for it to usurp the re- 
sources of another, less important task. In gen- 
eral, this process examines the task agenda from 
back to front until either (1) an acceptable re- 
source is found, in which case it is usurped and 
the two tasks are re-merged into the agenda, or 
(2) the process reaches a task with higher prior- 
ity than the intented usurper’s, in which case no 
suitable resource is available. 

Figure 3(a) shows the state of a work- 
cell as cylinder.1, in compartmenti , is to 
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Figure 3: CSR in action. 


be replaced with cylinder_4. The work- 
cell also contains a cylinder (cylinder^) op- 
erating in compartment 2 and scheduled for 
replacement by cylinder^ at some later 
time. The routinejreplace task associ- 
ated with compartment 1 acts, generating a 
deinstall_component task, which is placed at 
the head of the agenda (since it is primitive, 
and hence, immediately executable). This in 
turn issues a TLRPS put_at command to remove 
c y Under _1. Sometime during the execution of 
this command, cylinder_4 is removed, result- 
ing in the situation shown in Figure 3(b). At 
this point, since compartmenti is empty, and 
no longer has a reserved component, it gen- 
erates an await .component task. Cylinder^ 
should be “available” to this new task, but it 
is currently allocated to the routine .replace 
task associated with cdmpartment 2 . However, 
since the new task has higher priority, it usurps 
the resources (cylinder J}) of the lower priority 
routine_replace task. This changes its status 
to : executable, its action generates a primitive 
install-component task, which in turn issues a 
TLRPS put _in command to complete the replace- 
ment, as shown in Figure 3(c). 


3.3 CSR/TLRPS Integration 


The two systems presented above have been de- 
veloped jointly but at physically different sites 
in McLean, Virgina (CSR), and Houston, Texas 
(TLRPS). CSR is implemented in Portable Com- 
mon Loops (PCL) and resides on a Symbolics 
Lisp Machine. As noted above, TLRPS resides 
on a Silicon Graphics machine. The two systems 
are connected on an Ethernet LAN at the Hous- 
ton site. They communicate via TCP/IP streams 
over this network, using the interface language 
presented in Section 2.2. 
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3,4 The TLRPS Simulator 

Since CSR and TLRPS have been developed at 
different sites, a TLRPS simulator has been im- 
plemented for CSR’s development and testing. 
In addition to simulating TLRPS’s physical-space 
reasoning, allowing external agents to manip- 
ulate the domain, this simulator models com- 
ponent operation so that unexpected (i.e., pre- 
MTTF) component failures may occur. It also 
provides a graphic user-interface to CSR. 

The TLRPS simulator used in testing CSR also 
runs on a Symbolics Lisp Machine, with the two 
systems simulating CSR*-+TLRPS interface over a 
local Chaosnet. Figure 3 was generated using 
screen images from the TLRPS simulator. 

4 Current Status and Future 
Work 

This integrated project has addressed the prob- 
lem of integrating high-level and task-level rea- 
soning in a dynamic environment. The ar- 
chitectures used in both systems are domain- 
independent, and will be useful for other NASA 
applications, as well as broader application in 
manufacturing and assembly, hazardous materi- 
als handling, military operations, and undersea 
work. 

The existing system is able to react to any unex- 
pected change that occurs during the execution 
of a single CSR primitive task. During periods of 
CSR inactivity, updates are available at a rate of 
approximately one every ten seconds. CSR’s real 
response time is on the order of one-tenth of a 
second, so the “snapshot” nature of updates and 
update processing suffices for this domain. In 
general, the rate of change in dynamic domains 
is much faster, so that situated reasoning must 


be based on projecting future states in order to 
anticipate and avoid exception situations. An 
approach to situated reasoning based on these 
observations is presented in [San88] . 

Due to existing hardware, the current system 
has very little low-level reactive capability. An 
improved hardware system and more integrated 
reasoning and control architecture will be re- 
quired for more general purpose, robust au- 
tonomous control. To this end, MITRE is es- 
tablishing an Autonomous Systems Laboratory 
(ASL). Research in the ASL will focus on the in- 
tegration of deliberative (off-line) planning, sit- 
uated reasoning, and hardware subsystems. The 
ASL will be composed of “off the shelf” sensing, 
robotics, and AI hardware and firmware repre- 
senting the significant advances made in these 
technologies in recent years. The focus of the 
current project will shift from situated reasoning 
under a constant goal (e.g., routine repair and 
replace), toward more flexible control in a do- 
main where several different types of goals are 
to be achieved over time. A ground-based mo- 
bile system operating in a dynamic domain will 
be used as a test-bed to simulate a flexible space- 
based automaton for routine extra-vehicular re- 
pair, assembly, and retrieval tasks. Deliberative 
planning will take as input a collection of tasks to 
be carried out (the “daily schedule” ) and deter- 
mine an ordering among these tasks. Its output 
will be information used to monitor activity and 
constrain low-level task execution in the domain 
via a situated reasoning system. This latter sys- 
tem actually controls the physical system as it 
operates in its environment by controlling its re- 
actions to existing and anticipated states of af- 
fairs. This on-line system uses the constraints 
from the deliberative planner as heuristics in se- 
lecting among tasks it can perform, but is inde- 
pendently capable of a basic level of competence 
in the domain. 
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